All Questions
Tagged with message-queuequeue
7 questions
2votes
2answers
1kviews
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
1vote
1answer
1kviews
Pattern to limit message queue processing based on external API throttling
I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
- 167
6votes
2answers
773views
Design question for handling large volumes of messages in multi-tenant queue
I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based ...
- 169
3votes
3answers
301views
Dealing with data arriving at a different times
I have a load balanced service that needs two data inputs to correctly give a result. One of these inputs comes from a queue, the other comes from user input. Most of the time the data that comes ...
- 4,077
9votes
2answers
3kviews
Fair distributed task scheduling with RabbitMQ
Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
2votes
3answers
3kviews
Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?
My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
- 2,504
6votes
3answers
1kviews
Slow throughput - still worth using a dedicated message queue?
Here are some requirements for a queue: Every few days add ~100k tasks with various priorities Workers will pull tasks at typically less than 10 / second Tasks need to be completed by ~2 unique ...
- 163
